Speed learning on the fly

The practical performance of online stochastic gradient descent algorithms is highly dependent on the chosen step size, which must be tediously hand-tuned in many applications. The same is true for more advanced variants of stochastic gradients, such as SAGA, SVRG, or AdaGrad. Here we propose to adapt the step size by performing a gradient descent on the step size itself, viewing the whole performance of the learning trajectory as a function of step size. Importantly, this adaptation can be computed online at little cost, without having to iterate backward passes over the full data. Introduction This work aims at improving gradient ascent procedures for use in machine learning contexts, by adapting the step size of the descent as it goes along. Let `0, `1, . . . , `t, . . . be functions to be maximised over some parameter space Θ. At each time t, we wish to compute or approximate the parameter θ∗ t ∈ Θ that maximizes the sum Lt(θ) := ∑ s≤t `s(θ). (1) In the experiments below, as in many applications, `t(θ) writes `(xt, θ) for some data x0, x1, . . . , xt, . . . A common strategy, especially with large data size or dimensionality [Bot10], is the online stochastic gradient ascent (SG) θt+1 = θt + η ∂θ`t(θt) (2) with step size η, where ∂θ`t stands for the Euclidean gradient of `t with respect to θ. Such an approach has become a mainstay of both the optimisation and machine learning communities [Bot10]. Various conditions for convergence exist, starting with the celebrated article of Robbins and Monro [RM51], or later [KC78]. Other types of results are proved in convex settings, Several variants have since been introduced, in part to improve the convergence of the algorithm, which is much slower in stochastic than than in